{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 朴素贝叶斯\n",
    "\n",
    "> 理论 《统计学习方法》第四章 朴素贝叶斯法\n",
    "> \n",
    "> 代码 numpy version\n",
    ">\n",
    "> Python3.7"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "朴素贝叶斯基于贝叶斯定理与特征条件独立假设，用于分类。\n",
    "\n",
    "对训练集，基于特征条件独立的假设学习输入与输出的联合概率分布。然后使用贝叶斯定理计算出后验概率最大的输出作为分类结果"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "先验概率：$P(y=c_k), k=1,..,K$\n",
    "\n",
    "条件概率分布：$P(X=x|Y=c_k) = P(X^1=x^1,...,X^n=x^n|Y=c_k)$\n",
    "\n",
    "(k是类别数，n是特征数量)\n",
    "\n",
    "这儿假设**特征条件独立**，有$$P(X=x|Y=c_k) = P(X^1=x^1,...,X^n=x^n|Y=c_k) \\\\= \\Pi^n_j P(X^j=x^j|Y=c_k)$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 贝叶斯定理\n",
    "\n",
    "$$ 后验概率 ={ 先验概率 * 条件概率 \\over {事实} }$$\n",
    "\n",
    "贝叶斯定理：\n",
    "$$\n",
    "P(Y=c_k | X=x) = {P(X=x|Y=c_k)P(Y=c_k) \\over { P(X=x)}} \n",
    "$$\n",
    "\n",
    "全概率公式：\n",
    "$$\n",
    "P(X=x) = \\varSigma \\ P(Y)\\cdot P(X=x|Y)\n",
    "$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 朴素贝叶斯分类器\n",
    "$$\n",
    "y = f(x) = \\arg \\max {P(Y=c_k)\\cdot \\Pi^n_j P(X^j=x^j|Y=c_k) \\over (\\varSigma \\ P(Y=c_k)\\cdot\\Pi^n_j P(X^j=x^j|Y=c_k))} \n",
    "$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 参数估计\n",
    "\n",
    "使用极大似然估计，用数据集样本分布代表总体的分布\n",
    "\n",
    "先验概率$P(y=c_k)$的极大似然估计$\\varSigma I(y_i=c_k) \\over N$\n",
    "\n",
    "条件概率$P(X^1=x^j|Y=c_k)$的极大似然估计$\\varSigma I(X^j = x^j |y_i=c_k) \\over {\\varSigma I(y_i = c_k)}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.utils.data as Data\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "x_train = np.array([[1,0],[1,1],[1,1],[1,0],[1,0],[2,0],[2,1],[2,1],[2,3],[2,3],[3,3],[3,1],[3,1],[3,3],[3,3]])\n",
    "y_train = np.array([-1,-1,1,1,-1,-1,-1,1,1,1,1,1,1,1,-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1  1]\n",
      "[0.4 0.6]\n"
     ]
    }
   ],
   "source": [
    "# 计算先验概率\n",
    "y_unique = np.unique(y_train)\n",
    "print(y_unique)\n",
    "\n",
    "p_y = np.array([0 for i in range(len(y_unique))],dtype=float)\n",
    "for i,v in enumerate(y_unique):\n",
    "    p_y[i] = len(y_train[y_train == v]) / len(y_train)\n",
    "print(p_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [0 1 3]]\n"
     ]
    }
   ],
   "source": [
    "x_unique = np.array([np.unique(x_train[:,i]) for i in range(x_train.shape[1])])\n",
    "print(x_unique)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 3]]\n",
      "P(x^1=1|Y=-1) 3/6\n",
      "P(x^1=2|Y=-1) 2/6\n",
      "P(x^1=3|Y=-1) 1/6\n",
      "P(x^2=0|Y=-1) 3/6\n",
      "P(x^2=1|Y=-1) 2/6\n",
      "P(x^2=3|Y=-1) 1/6\n",
      "P(x^1=1|Y=1) 2/9\n",
      "P(x^1=2|Y=1) 3/9\n",
      "P(x^1=3|Y=1) 4/9\n",
      "P(x^2=0|Y=1) 1/9\n",
      "P(x^2=1|Y=1) 4/9\n",
      "P(x^2=3|Y=1) 4/9\n",
      "[[[0.5        0.22222222]\n",
      "  [0.33333333 0.33333333]\n",
      "  [0.16666667 0.44444444]]\n",
      "\n",
      " [[0.5        0.11111111]\n",
      "  [0.33333333 0.44444444]\n",
      "  [0.16666667 0.44444444]]]\n"
     ]
    }
   ],
   "source": [
    "# 需要一个三维数组进行存储\n",
    "# p[i][j][k] i是x的第i个特征，j是第i个特征的第j个取值,k是y的第k个取值\n",
    "x_feature_count = 3\n",
    "p_matrix = np.zeros((len(x_unique),x_feature_count,len(y_unique)))\n",
    "\n",
    "print(x_train[ (y_train == -1) &  (x_train[:,0] == 3) ])\n",
    "for k,vk in enumerate(y_unique):\n",
    "    for i,vi in enumerate(x_unique):\n",
    "        for j,vj in enumerate(x_unique[i]):\n",
    "            top_count = len(x_train[ (y_train == vk) &  (x_train[:,i] == vj) ])\n",
    "            bottom_count =  len(y_train[y_train == vk])\n",
    "            print('P(x^{}={}|Y={})'.format(i+1,vj,vk),end=' ')\n",
    "            print(top_count,end='/' )\n",
    "            print(bottom_count)\n",
    "            p_matrix[i][j][k] = top_count / bottom_count\n",
    "\n",
    "print(p_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.06666666666666667, 0.02222222222222222]\n",
      "simple [2, 0] after nativBayes predict result is -1\n"
     ]
    }
   ],
   "source": [
    "def CalResP(x,y_k):\n",
    "    p = p_y[y_k]\n",
    "    for i in range(len(x)):\n",
    "        j,k = np.where(x_unique[i]==x[i])[0][0],y_k\n",
    "        p *= p_matrix[i][j][k]\n",
    "    return p\n",
    "\n",
    "def CalCla(x):\n",
    "    p_res = [0 for i in range(len(y_unique))]\n",
    "    for i,y_k in enumerate(y_unique):\n",
    "        p_res[i] = CalResP(x,i)\n",
    "    print(p_res)\n",
    "    return y_unique[np.argmax(np.array(p_res))]\n",
    "\n",
    "s = [2,0]\n",
    "print('simple {} after nativBayes predict result is {}'.format(s,CalCla(s)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "3c874ee3d65450ef899cacaa06bdca40f9e4450c5e631a94911db3215dbda5b8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
